\[\begin{align*} {\mathbf B}_{3\times 4} = \left(\begin{array}{cccc} b_{11} & b_{12} & b_{13} & b_{14}\\ b_{21} & b_{22} & b_{23} & b_{24}\\ b_{31} & b_{32} & b_{33} & b_{34}\\ \end{array} \right) \end{align*}\]

then

\[\begin{align*} {\mathbf A}{\mathbf B}_{2\times 4} = \left(\begin{array}{cccc} \sum_{j=1}^3 a_{1j}b_{j1} & \sum_{j=1}^3 a_{1j}b_{j2} & \sum_{j=1}^3 a_{1j}b_{j3} & \sum_{j=1}^3 a_{1j}b_{j4}\\ \sum_{j=1}^3 a_{2j}b_{j1} & \sum_{j=1}^3 a_{2j}b_{j2} & \sum_{j=1}^3 a_{2j}b_{j3} & \sum_{j=1}^3 a_{2j}b_{j4} \end{array} \right) \end{align*}\]

Pour the rows into the columns.

Note: You can’t do \({\mathbf B}{\mathbf A}\)!

This is the matrix equivalent of 1

\(AI = A\) and \(IA = A\) (as long as \(A\) was conformable with these matrix multiplications)

Therefore we split the data and conduct different tasks on different subsets of data

DO NOT only evaluate your model on the training set.

The model was estimated on that training set so it is designed to predict that set well.

Therefore this will give you a BIASED estimate of its performance.

In particular a more complex model will always APPEAR to predict the training set better

Let \(\widehat{y}:\mathcal{X}\mapsto\mathcal{Y}\) be the model estimated from training data, \(\{(y_i, {\mathbf x}_i)\}_{i = 1}^n\).

Consider a classification problem i.e. \(\mathcal{Y} = \{1, \ldots, K\}\), a number for each class.

We can evaluate this model using the error rate (fraction of misclassifications)

We can look at this in the training data, \(\{(y_i, {\mathbf x}_i)\}_{i = 1}^n\) to get the Training Error Rate

\[\text{Error rate} = \frac{1}{n}\sum_{i=1}^n I(y_i \ne \widehat{y}({\mathbf x}_i))\]

But, a better estimate of future accuracy is obtained using test data to get the Test Error Rate.

In classification, the majority of machine learning models will not be a functions that produces prediction - \(\widehat{y}:\mathcal{X}\mapsto\{1, \ldots, K\}\)

We will see in this class that it is much better to estimate probabilities of being in each class.

Let these probabilities be \(\widehat{p}:\mathcal{X}\mapsto (0, 1)^K\) with \(\sum_{k=1}^K\widehat{p}(x_i)_k = 1\) where \(\widehat{p}(x_i)_k = \mathbb{P}(y = k\mid x_i)\)

We can always turn these probabilities into class prediction i.e. \(\widehat{y}(x_i) = \arg\max \widehat{p}(x_i)_k\)

But the probabilities themselves contain much more information than just a class labels

And it is easier to train a model to learn them

Consider binary classification, i.e. \(\mathcal{Y} = \{0, 1\}\)

Then a decision/prediction rule would be to set \(\hat{y}(x_i) = 1\) if \(\hat{p}(x_i) > \alpha\)

One logical value for \(\alpha = 0.5\)

However if the probability that a relative has cancer is 0.45, do you want the doctor to predict they don’t have cancer??

The value of \(\alpha\) is problem specific and depends on the consequences of missclassification.

It is not our job as data scientists to decide on \(\alpha\), we model the probabilities and let a domain expert turn this into a decision.

The ROC curve allows you to evaluate the accuracy for all values of \(\alpha\)

Black line is true boundary.

Goal: Without knowing what the true structure is, fit the signal and ignore the noise. Be flexible but not too flexible.

Other times you can get the same results by transforming the original variables (See tidymodels steps.)